Dynamic in-flight database request throttling

ABSTRACT

Various embodiments herein each include at least one of systems, methods and software for dynamic in-flight database request throttling. One such embodiment includes monitoring a database or multi-database system Workload Definition (WD) to identify queuing of a higher priority request while the database system is processing requests of a lower priority and when queuing of a higher priority request is identified, adjusting a metric throttle for the WD to a new metric throttle level C n , computed as the average of a metric level C c  that would drive the metric to a target T and a metric level C r  that would drive a rolling average of the metric to the target T. This embodiment further includes evaluating lower priority in-flight requests for the current workload to identify abort candidate requests to abort and aborting the identified abort candidate requests and place the aborted requests in a delay queue for later execution. Some embodiments of this method also include, when the metric T has not yet been met, identifying lower priority in-flight requests of the current workload to identify suspend candidate requests and suspending the identified suspend candidate requests and place the suspended requests in a suspend queue to complete execution later.

PRIORITY APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication Ser. No. 62/272,221, filed Dec. 29, 2015, the content ofwhich is hereby incorporated by reference in its entirety.

BACKGROUND INFORMATION

Data can be an abstract term. In the context of computing environmentsand systems, data can generally encompass all forms of informationstorable in a computer readable medium (e.g., memory, hard disk). Data,and in particular, one or more instances of data can also be referred toas data object(s). As is generally known in the art, a data object can,for example, be an actual instance of data, a class, a type, or aparticular form of data, and so on.

Data can be an abstract term. In the context of computing environmentsand systems, data can generally encompass all forms of informationstorable in a computer readable medium (e.g., memory, hard disk). Data,and in particular, one or more instances of data can also be referred toas data object(s). As is generally known in the art, a data object can,for example, be an actual instance of data, a class, a type, or aparticular form of data, and so on.

Generally, one important aspect of computing and computing systems isstorage of data. Today, there is an ever increasing need to managestorage of data in computing environments. Databases are good examplesof computing environments or systems where the storage of data can becrucial. As such, databases are discussed below in greater detail as anexample.

The term database can also refer to a collection of data and/or datastructures typically stored in a digital form. Data can be stored in adatabase for various reasons and to serve various entities or “users.”Generally, data stored in the database can be used by one or more of the“database users.” A user of a database can, for example, be a person, adatabase administrator, a computer application designed to interact witha database, etc. A very simple database or database system can, forexample, be provided on a Personal Computer (PC) by storing data (e.g.,contact information) on a Hard Disk and executing a computer programthat allows access to the data. The executable computer program can bereferred to as a database program, or a database management program ordatabase management system. The executable computer program can, forexample, retrieve and display data (e.g., a list of names with theirphone numbers) based on a request submitted by a person (e.g., show methe phone numbers of all my friends in Ohio).

Generally, database systems are much more complex than the example notedabove. In addition, databases have been evolved over the years and areused in various business and organizations (e.g., banks, retail stores,governmental agencies, universities). Today, databases can be verycomplex. Some databases can support several users simultaneously andallow them to make very complex queries (e.g., give me the names of allcustomers under the age of thirty-five (35) in Ohio that have bought allthe items in a given list of items in the past month and also havebought a ticket for a baseball game and purchased a baseball hat in thepast 10 years).

Typically, a Database Manager (DBM) or a Database Management System(DBMS) is provided for relatively large and/or complex databases. Asknown in the art, a DBMS can effectively manage the database or datastored in a database, and serve as an interface for the users of thedatabase. For example, a DBMS can be provided as an executable computerprogram (or software) product as is also known in the art.

It should also be noted that a database can be organized in accordancewith a Data Model. Some notable Data Models include a Relational Model,an Entity-relationship model, and an Object Model. The design andmaintenance of a complex database can require highly specializedknowledge and skills by database application programmers, DBMSdevelopers/programmers, database administrators (DBAs), etc. To assistin design and maintenance of a complex database, various tools can beprovided, either as part of the DBMS or as free-standing (stand-alone)software products. These tools can include specialized Databaselanguages (e.g., Data Description Languages, Data ManipulationLanguages, Query Languages). Database languages can be specific to onedata model or to one DBMS type. One widely supported language isStructured Query Language (SQL) developed, by in large, for RelationalModel and can combine the roles of Data Description Language, DataManipulation Language, and a Query Language.

Today, databases have become prevalent in virtually all aspects ofbusiness and personal life. Moreover, usage of various forms ofdatabases is likely to continue to grow even more rapidly and widelyacross all aspects of commerce, social and personal activities.Generally, databases and DBMS that manage them can be very large andextremely complex partly in order to support an ever increasing need tostore data and analyze data. Typically, larger databases are used bylarger organizations, larger user communities, or device populations.Larger databases can be supported by relatively larger capacities,including computing capacity (e.g., processor and memory) to allow themto perform many tasks and/or complex tasks effectively at the same time(or in parallel). On the other hand, smaller database systems are alsoavailable today and can be used by smaller organizations. In contrast tolarger databases, smaller databases can operate with less capacity.

A current popular type of database is the relational database with aRelational Database Management System (RDBMS), which can includerelational tables (also referred to as relations) made up of rows andcolumns (also referred to as tuples and attributes). In a relationaldatabase, each row represents an occurrence of an entity defined by atable, with an entity, for example, being a person, place, thing, oranother object about which the table includes information.

One important objective of databases, and in particular a DBMS, is tooptimize the performance of queries for access and manipulation of datastored in the database. Given a target environment, an “optimal” queryplan can be selected as the best option by a database optimizer (oroptimizer). Ideally, an optimal query plan is a plan with the lowestcost (e.g., lowest response time, lowest CPU and/or I/O processing cost,lowest network processing cost). The response time can be the amount oftime it takes to complete the execution of a database operation,including a database request (e.g., a database query) in a given system.In this context, a “workload” can be a set of requests, which mayinclude queries or utilities, such as, load that have some commoncharacteristics, such as, for example, application, source of request,type of query, priority, response time goals, etc.

Generally, data (or “Statistics”) can be collected and maintained for adatabase. “Statistics” can be useful for various purposes and forvarious operational aspects of a database. In particular, “Statistics”regarding a database can be very useful in optimization of the queriesof the database, as generally known in the art.

More recently, in-memory processing systems, including in-memorydatabase systems have been developed where data is typically stored andprocessed in memory which can offer much faster processing times thansystems that also store data for processing in non-volatile orpersistent storages (e.g., Hard Disk Drives (HDD's, Solid Disk Drives(SOD), Flash memory).

Database systems and environments are useful.

SUMMARY

Various embodiments herein each include at least one of systems, methodsand software for dynamic in-flight database request throttling. One suchembodiment includes monitoring a database system or multi-databasesystems Workload Definition (WD) to identify queuing of a higherpriority request while the database system(s) are processing requests ofa lower priority and when queuing of a higher priority request isidentified, adjusting a metric throttle for the WD to a new metricthrottle level C_(n), computed as the average of a theoretical metriclevel C_(c) that would drive the metric to a target T and a theoreticalmetric level C_(r) that would drive a rolling average of the metric tothe target T. This embodiment further includes evaluating lower priorityin-flight requests for the current workload to identify abort candidaterequests to abort and aborting the identified abort candidate requestsand place the aborted requests in a delay queue for later execution.Some embodiments of this method also include, when the metric T has notyet been met, identifying lower priority in-flight requests of thecurrent workload to identify suspend candidate requests and suspendingthe identified suspend candidate requests and place the suspendedrequests in a suspend queue to complete execution later.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows the effect of managing with a rolling average.

FIG. 2 shows a comparison between a current metric and a rolling averageof the metric.

FIG. 3 is a flow chart.

FIG. 4 is a flow chart.

DETAILED DESCRIPTION

As noted in the background section, database systems and environmentsare useful.

Currently, database management systems can monitor and effectivelycontrol processing of database queries. For example, Teradata WorkManagement (TDWM) “Traffic Cop” can let a user choose an event type fora Workload Definition (WD) defined in a RuleSet. This is commonlyreferred to as a: “By-WD” event. Today, these events can be usedprimarily for monitoring and reporting purposes, to gauge the success ofthe workload's performance, and to note trends with respect to meetingSLGs.

A second use of the By-WD events is to automatically detect when Arrivaland Concurrency levels are too high or conversely too low. For example,one of the primary approaches used by DBAs and System Administrators isto first identify that there is a problem with their system.Investigations into why will typically start with analysis at thesystem-level (System CPU). If the system is not 100% busy and does nothave heavy skewing, then typically the DBA can next check for:

-   -   a) if Arrival Rate>Throughput System Level Goals (SLG), then a        possible cause of the missed SLG is System Over-load. Not only        is the system falling behind and unable to keep up with arrivals        to this workload, other competing workloads may be impacting the        ability to at least deliver the throughput SLG;    -   b) if Arrival Rate<=Throughput SLG, then the cause of the missed        SLG is under-demand. In other words, there is insufficient        demand from the application servers to meet the throughput SLG.        The system could be nearly idle and still miss the throughput        SLG, so by pre-qualifying the missed SLG throughput event with        arrivals>throughput SLG, you avoid detecting an uninteresting        situation.

However, if the CPU is 100% busy, then the number of active sessionswill be checked for unusually high levels of concurrency. Concurrencycan be defined based on the concurrent requests or queries (e.g., two(2) concurrent SQL queries running at the same time).

Generally, these investigations are typically triggered based on theBy-WD event enabling the DBA to act manually or automatically to resolvethe situation, and bring WD performance back to SLG conformance. Toautomate correction for these type problems currently requires what iscalled a “state change”. A state change is a relatively expensiveoperation. Performing a state change in the middle of a systemundergoing workload management performance issues can have latencyissues to take affect due to the expense of the state change operation.It should be noted that customers cannot define ‘states’ for allscenarios without creating a large and complex State Matrix, i.e., apredefined set of Teradata Active System Management (TASM) rules forvaried conditions. Some customers go to the trouble to dynamicallycreate rule sets that adjust the throttle when a performance crisisoccurs. There is a need for dynamic methods to adjust throttles withouta ‘State Change’. In other words, “Dynamic throttles” are needed.

Dynamic throttles as defined today provide a mechanism to put limits onthe number of new requests, such as SQL requests, by workloadclassification. Such limits can be dynamically adjusted by the systembased upon an assessment of the system's ability to take on new workbased upon an historical rolling average of resource availability andestimates of resource requirements for the type of requests thethrottles apply. The throttles provide a mechanism to prevent new workfrom entering the system if that new work would over utilize the systemresources (CPU, memory, AWTs, etc.) and thus cause performancedegradation and missed SLAs.

In one aspect, a method for implementing Dynamic Throttles is provided.The methods can, for example, be provided as a light-weight method forimplementing Dynamic throttles that can solve problems associatedconventional techniques with relatively less overhead. In doing so, atype of dynamic throttle (concurrency) event can be defined to assistthe customers in managing complex workload management by environments,automatically changing the value of a throttle as an event. A new typeof event (throttle event) that can automatically act within theframework of the TDWM regulator but does not require a State change.

It should be noted that the technique can also be applied to unusuallyhigh numbers of AMP Worker Task (AWT) activity. If some workloads havetoo many active sessions, then appropriate actions can be taken, forexample, to limit concurrency (with a throttle), to abort or suspendqueries, and/or to make adjustments to the Priority Scheduler weights.If the CPU is 100% busy and active sessions looks ok, the DBA might nextcheck the CPU usage by WD and/or Session to see if there is a runawayquery. From here the DBA can take the appropriate action, usually toabort or suspend the offending request or move it to a Penalty-box witha CPU limit or cap.

Often the business' ultimate management goal is to manage a workload'sconcurrency on an hourly or daily basis, without concern for momentarylow or high-usage. As such, the customers desire the opportunity to makeup for low-usage moments by over-consuming for a time. Those low-usagemoments can be due to either low Timeshare issues, or under-demand.Therefore, a new option is created for Timeshare only WDs.

By-WD events allow the DBA to manage based on a Rolling Averaging 102whose duration is of the DBA's choosing, as shown in FIG. 1. TDWM wouldmanage the rolling average 102 and derive the resultant “Dynamicthrottle” value for the moment via control theory and other Statisticalprocessing techniques (SPC). It would then communicate a revisedconcurrency value to TDWM for its management at appropriate intervals.

TDWM can, for example, monitor key health and demand metrics (By-WDevents) (block 302) to determine what the timeshare WD throttle limitshould be at any given point in time (block 304), as shown in FIG. 3.The limit can change dynamically and constantly, just as the systemdemand and health characteristics also change constantly due to normalmixed workload variations. By monitoring these metrics and thenadjusting the throttle limit based on those metrics, the key health anddemand resources can be indirectly monitored to stay within healthylevels, thereby maintaining the system in a healthier state.

The monitoring of key metrics and subsequent throttle limit adjustmentcan rely on control theory and Statistical processing techniques toreduce oscillation in the regulation (“cruise-control”). The controltheory technique used to accomplish this is to base adjustments on boththe current actual metric values as well as the historical metricvalues. A technique for dynamic automatic throttles can include thefollowing:

-   -   Target Value (T) for each monitored metric. (Concurrency, AWTs,        CPU, I/O, Memory, Service Time delays, etc.)    -   Data obtained from monitoring the By-WD events is done        internally within the TDWM regulator:        -   Current metric value (Cur)        -   Current Timeshare Automatic Throttle Limit (C_(c))        -   Current Timeshare Concurrency Level (A) (for “active”            concurrent queries        -   RollingAvg Timeshare Concurrency (A_(r))        -   RollingAvg metric value (Cur_(r))

Some embodiments include performing the following analysis andconcurrency throttle adjustment at every Traffic Cop event interval, asshown in FIG. 4.

(For example, every 60, 600, 3600 seconds, etc.)

-   -   Derive theoretical concurrency level (C_(c)) that would drive        metrics back to target (block 402):

C _(c)=(T*A)/Cur

-   -   Derive theoretical concurrency level (C_(r)) that would drive        Rolling Average of the Concurrency back to target (block 402):

C _(r)=(T*A _(r))/Cur_(r)

-   -   Adjust the WD's concurrency throttle to the average 1 of C_(c)        and C_(r) (block 406):

C _(n)=AVG(C _(c) ,C _(r))

-   -   Adjust for all metrics, with the resulting throttle limit to        impose based on the most restrictive of all metrics analyzed        (block 408):

C _(n)=min(C ₁ ,C ₂,etc.)

-   -   Final adjustments: subject to maximum limit heuristics—For        example, Timeshare limit cannot exceed X % of total AWT pool        (block 410).    -   Determine if any queries in the TDWM queue can be released based        on the new throttle value (block 412). If so dynamically change        the throttle value and release the queries from the TDWM queue.

Min keeps the metric below the threshold. Average would allowover-consumption to compensate for under-consumption, which is usefulfor concurrency management.

FIG. 2 shows an environment that was allowed to run with no concurrencythrottle values established for a small period of time. Here you can seethe wild swings in the current metric (concurrency) levels 202. In thisexample a Target (T) concurrency value 204 was established by using aRolling average of 3600 seconds. In this particular case you can see aTarget (T) throttle value of 12 was derived. The current concurrency 202and rolling average metrics 206 is measured as a By-WD event with atimer set at 60 seconds. The task of the dynamic concurrency throttlealgorithm is to bring the environment's concurrency health metrics backto target. FIG. 2 shows the result of the algorithm described above inbringing the concurrency back in line with the target. It is importantto note that a goal of some embodiments is to bring the current metricsback to conformity and then keep it there. FIG. 2 demonstrates thisalgorithm's ability to do this effectively. Note that because actualactive concurrency is part of the formula, it doesn't necessarily adjusta limit up when there is no demand.

This approach can simplify the implementation of dynamic throttles aswell as removing the dependency of a State change at the cost of usercomplexity.

However, one issue with this approach is that in-flight, or in process,requests are not dynamically suspended or aborted. Thus, if a rush ofhigher priority work arrives that should take precedent over the currentwork being processed and causes a dynamic throttle to allow lessconcurrency on a particular workload, that workload's concurrency limitwill be exceeded until enough in-flight requests complete to drop thecurrency level down to its new concurrency limit. This allows thatworkload to consume resources that could, and likely should, beavailable to the newly requested higher priority work. The net effect ofthis scenario is that although the system will eventually reach adynamically changed throttle limit, the system is slower than ideal inreacting to a dynamic throttle limit change to reduce concurrency.

Thus, dynamic throttles in some embodiments take into considerationin-flight requests in addition to new incoming requests. Two possibleactions or methods can be used for reducing the concurrency until theworkload throttle concurrency level is met. One method is to ‘silently’abort a request and place a new instance of it on the workloadmanagement delay queue to be run later. This method to abort a requestand restart it from the first execution step is called the Abort method.The second method is to ‘suspend’ a request at a quiet point followingcompletion of a unit of work of the request and place it on the delayqueue to be resumed later. This method is called the Suspend method.Requests placed on the delay queue using either the Abort or Suspendmethod would be released from the delay queue using the existing TASMmethodology for releasing a request. In this case, it would be eitherwhen the concurrency level for requests in that workload drops below thenew level set by the dynamic throttle, or when the dynamic throttleloosens the concurrency limit allowing more concurrency. Note that insome embodiments, when requests are released, requests that have beensuspended are released prior to requests that have been aborted.

The Abort method has the advantage of reaching the concurrency levelquicker, at a cost of re-executing the completed portion of work. TheSuspend method has the opposite effect, that is, no work is lost, butwhen compared to the Abort method there will still be some delay (inorder to complete the in-flight step and to cache data representative ofthe work in progress and the suspended state of the request) before arequest can be suspended. Additionally, requests put on the delay queuethrough the Abort method hold no locks or spool resources. Requestsdelayed using the Suspend method may continue to hold locks and haveintermediate results that must be retained. To avoid potential blockingissues, some embodiments include an option that limits Suspend methodcandidate requests to a specific lock severity level (e.g., only requestusing Access locks be considered).

Upon adjusting the dynamic throttle limit lower for a given workload inorder to reduce concurrency, TASM would first evaluate in-flightrequests for that workload to identify Abort Method candidates. Thefirst step in some embodiments is to identify requests for the Abort andtake action of those requests. After processing the Abort Methodcandidates, if additional adjustments are needed, TASM would nextevaluate in-flight requests for that workload to identify Suspend Methodcandidates, and then take action on those requests.

Requests may be chosen as Abort method candidates when their estimatedprogress (based on the last completed step) is less than a ‘percentagecomplete’ setting. The ‘percentage complete’ value may be userconfigurable and default to a system-provided value (for example, 10°%). Different dynamic throttles could have different ‘percentagecomplete’ settings. Note this user configurable setting allows a userthe flexibility so that either only the Abort method is used or thatonly the Suspend method is used. That is, a user configurable setting of0% would indicate no requests would use the Abort method, and a userconfigurable setting of 100% would indicate all requests would use theAbort method.

In the Abort method phase, if reducing workload's concurrency by thenumber of candidates does not bring the concurrency to the desired valueor brings it exactly to the desired value, then all candidates would besilently aborted by TASM and placed back onto the delay queue to beexecuted from the first execution step using normal TASM delay queuerequest release algorithms, as is done today when a request is put onthe delay queue. If reducing a workload's concurrency by the number ofcandidates would bring the concurrency below the desired value, thenonly a subset of those candidates needs to be chosen and the Abortmethod applied. There are various methods for choosing the order toselect the subset of candidates to Abort. For example, they may bechosen by applying a Last-In-First-Out (LIFO) algorithm, the requestswhich had the least estimated percentage complete, based on a determinedprocessing cost of the various requests and a fitting algorithm thatselects a combination of the requests with a sum cost that will achievea desired concurrency level, among other possible algorithms.

In some embodiments, when the Abort method was successful in reachingthe desired concurrency level, then the Suspend method would not beused.

In some embodiments, after completing the Abort method phase, when thedesired concurrency level for the dynamic throttle was not reached, TASMwould next identify requests for the Suspend method. N requests areidentified for suspension, where N is the number of requests needed tomeet the desired concurrency level for the workload after the Abortmethod phase completes. As with identifying candidate requests to beaborted, there are also various methods that may be utilized foridentifying requests to be suspended. Once identified, the requestswould continue normally until the completion of the step currently inprogress. Requests are typically broken into a number of steps forexecution by a request optimizer. These steps are generally units ofwork to be completed in performance of the request. Thus, when a currentunit of work completes, the request is then suspended and the next unitof work is not yet performed, data work-in-progress and status of therequest is cached and moved to the TASM delay queue. In someembodiments, when an in-flight request in the dynamic workload completesduring the time that TASM is waiting for candidate requests to Suspend,TASM will remove a request from the Suspend candidate list instead ofreleasing a request from delay queue. This is done until there are nomore Suspend candidates.

When releasing requests from the delay queue, TASM would release‘suspended’ requests prior to starting any new requests. Abortedrequests are the next priority, ahead of other requests delayednormally, i.e., where request execution had never started.

Such embodiments provide dynamic throttling benefit more by quicklyadjusting the system to adhere to the dynamic throttle changes, whichmore quickly allocates system resources to higher priority work upon itsarrival.

Generally, various aspects, features, embodiments or implementations ofthe invention described above can be used alone or in variouscombinations. Furthermore, implementations of the subject matter and thefunctional operations described in this specification can be implementedin digital electronic circuitry, or in computer software, firmware, orhardware, including the structures disclosed in this specification andtheir structural equivalents, or in combinations of one or more of them.Implementations of the subject matter described in this specificationcan be implemented as one or more computer program products, i.e., oneor more modules of computer program instructions encoded on a computerreadable medium for execution by, or to control the operation of, dataprocessing apparatus. The computer readable medium can be amachine-readable storage device, a machine-readable storage substrate, amemory device, a composition of matter affecting a machine-readablepropagated signal, or a combination of one or more of them. The term“data processing apparatus” encompasses all apparatus, devices, andmachines for processing data, including by way of example a programmableprocessor, a computer, or multiple processors or computers. Theapparatus can include, in addition to hardware, code that creates anexecution environment for the computer program in question, e.g., codethat constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, or a combination of one or moreof them. A propagated signal is an artificially generated signal, e.g.,a machine-generated electrical, optical, or electromagnetic signal thatis generated to encode information for transmission to suitable receiverapparatus.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, and it can bedeployed in any form, including as a standalone program or as a module,component, subroutine, or other unit suitable for use in a computingenvironment. A computer program does not necessarily correspond to afile in a file system. A program can be stored in a portion of a filethat holds other programs or data (e.g., one or more scripts stored in amarkup language document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, subprograms, or portions of code). A computer programcan be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read only memory ora random access memory or both. The essential elements of a computer area processor for performing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto-optical disks, or optical disks. However, a computerneed not have such devices. Moreover, a computer can be embedded inanother device, e.g., a mobile telephone, a personal digital assistant(PDA), a mobile audio player, a Global Positioning System (GPS)receiver, to name just a few. Computer readable media suitable forstoring computer program instructions and data include all forms ofnon-volatile memory, media and memory devices, including by way ofexample semiconductor memory devices, e.g., EPROM, EEPROM, and flashmemory devices; magnetic disks, e.g., internal hard disks or removabledisks; magneto optical disks; and CDROM and DVD-ROM disks. The processorand the memory can be supplemented by, or incorporated in, specialpurpose logic circuitry.

To provide for interaction with a user, implementations of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech,tactile or near-tactile input.

Implementations of the subject matter described in this specificationcan be implemented in a computing system that includes a backendcomponent, e.g., as a data server, or that includes a middlewarecomponent, e.g., an application server, or that includes a frontendcomponent, e.g., a client computer having a graphical user interface ora Web browser through which a user can interact with an implementationof the subject matter described in this specification, or anycombination of one or more such backend, middleware, or frontendcomponents. The components of the system can be interconnected by anyform or medium of digital data communication, e.g., a communicationnetwork. Examples of communication networks include a local area network(“LAN”) and a wide area network (“WAN”), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

While this specification contains many specifics, these should not beconstrued as limitations on the scope of the disclosure or of what maybe claimed, but rather as descriptions of features specific toparticular implementations of the disclosure. Certain features that aredescribed in this specification in the context of separateimplementations can also be implemented in combination in a singleimplementation. Conversely, various features that are described in thecontext of a single implementation can also be implemented in multipleimplementations separately or in any suitable sub-combination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asub-combination or variation of a sub-combination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the implementations described above should not beunderstood as requiring such separation in all implementations, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

It will be readily understood to those skilled in the art that variousother changes in the details, material, and arrangements of the partsand method stages which have been described and illustrated in order toexplain the nature of the inventive subject matter may be made withoutdeparting from the principles and scope of the inventive subject matteras expressed in the subjoined claims.

What is claimed is:
 1. A method comprising: monitoring at least onedatabase system Workload Definition (WD) to identify queuing of a higherpriority request while the at least one database system is processingrequests of a lower priority; when queuing of a higher priority requestis identified, adjusting a metric throttle for the WD to a new metricthrottle level C_(n); evaluating lower priority in-flight requests forthe current workload to identify abort candidate requests to abort;aborting the identified abort candidate requests.
 2. The method of claim1, further comprising: placing the aborted requests in a delay queue forlater execution.
 3. The method of claim 2, further comprising: when themetric T has not yet been met, identifying lower priority in-flightrequests of the current workload to identify suspend candidate requests;and suspending the identified suspend candidate requests; and placingthe suspended requests in a suspend queue to complete execution later.4. The method of claim 1, wherein the new metric throttle level C_(n) iscomputed as the average of a metric level C_(c) that would drive themetric to a target T and a metric level C_(r) that would drive a rollingaverage of the metric to the target T.
 5. The method of claim 4, whereinthe metric levels C_(c) and C_(r) are theoretical metric levels.
 6. Themethod of claim 1, wherein the at least one database system is amulti-database system.
 7. A non-transitory computer readable storagemedium with instructions stored thereon which when executed by acomputer processor cause the computer to perform data processingactivities comprising: monitoring at least one database system WorkloadDefinition (WD) to identify queuing of a higher priority request whilethe at least one database system is processing requests of a lowerpriority; when queuing of a higher priority request is identified,adjusting a metric throttle for the WD to a new metric throttle levelC_(n); evaluating lower priority in-flight requests for the currentworkload to identify abort candidate requests to abort; aborting theidentified abort candidate requests.
 8. The non-transitory computerreadable storage medium of claim 7, the data processing activitiesfurther comprising: placing the aborted requests in a delay queue forlater execution.
 9. The non-transitory computer readable storage mediumof claim 8, the data processing activities further comprising: when themetric T has not yet been met, identifying lower priority in-flightrequests of the current workload to identify suspend candidate requests;and suspending the identified suspend candidate requests, and placingthe suspended requests in a suspend queue to complete execution later.10. The non-transitory computer readable storage medium of claim 7,wherein the new metric throttle level C_(n) is computed as the averageof a metric level C_(c) that would drive the metric to a target T and ametric level C_(r) that would drive a rolling average of the metric tothe target T.
 11. The non-transitory computer readable storage medium ofclaim 10, wherein the metric levels C_(c) and C_(r) are theoreticalmetric levels.
 12. The non-transitory computer readable storage mediumof claim 7, wherein the at least one database system is a multi-databasesystem.
 13. A system comprising: at least one computer processor, atleast one network interface device, and at least one memory device;instructions stored on the at least one memory device that areexecutable by the at least one computer process to perform dataprocessing activities comprising: monitoring at least one databasesystem Workload Definition (WD) to identify queuing of a higher priorityrequest while the at least one database system is processing requests ofa lower priority; when queuing of a higher priority request isidentified, adjusting a metric throttle for the WD to a new metricthrottle level C_(n); evaluating lower priority in-flight requests forthe current workload to identify abort candidate requests to abort;aborting the identified abort candidate requests.
 14. The system ofclaim 13, the data processing activities further comprising: placing theaborted requests in a delay queue for later execution.
 15. The system ofclaim 14, the data processing activities further comprising: when themetric T has not yet been met, identifying lower priority in-flightrequests of the current workload to identify suspend candidate requests;and suspending the identified suspend candidate requests; and placingthe suspended requests in a suspend queue to complete execution later.16. The system of claim 13, wherein the new metric throttle level C_(n)is computed as the average of a metric level C_(c) that would drive themetric to a target T and a metric level C_(r) that would drive a rollingaverage of the metric to the target T.
 17. The system of claim 16,wherein the metric levels C_(c) and C_(r) are theoretical metric levels.18. The system of claim 13, wherein the at least one database system isa multi-database system.